feat: simplify contradicting and redundant predicates on a column by wudidapaopao · Pull Request #25207 · apache/datafusion

wudidapaopao · 2026-09-11T20:22:29Z

Which issue does this PR close?

Closes Simplify predicate expressions like 'a > 1 and a < 1' to constant false #5830.

Rationale for this change

simplify_predicates reduces the >/>= and the </<= comparisons on a column to their most restrictive bound, but never compares the two groups with each other, and only looks for contradictions between equalities. A filter no row can satisfy therefore still scans the table.

`WHERE` clause	Before	After
`a > 3 AND a < 1`	`a > 3 AND a < 1`	`EmptyRelation`
`a > 1 AND a < 1`	`a > 1 AND a < 1`	`EmptyRelation`
`a >= 1 AND a < 1`	`a >= 1 AND a < 1`	`EmptyRelation`
`a = 7 AND a < 2`	`a = 7 AND a < 2`	`EmptyRelation`
`a = 7 AND a != 7`	`a = 7 AND a != 7`	`EmptyRelation`
`a = 7 AND a > 5`	`a = 7 AND a > 5`	`a = 7`
`a > 10 AND a != 5`	`a > 10 AND a != 5`	`a > 10`

a >= 1 AND a <= 1 still simplifies to a = 1: bounds meeting at a value both admit stay satisfiable.

What changes are included in this PR?

fix: skip comparisons against a NULL literal when grouping. ScalarValue::try_cmp follows sort order, where NULL sits below every other value, so a > NULL AND a > 5 was reduced to a > 5 although a > NULL is never true. Reachable only through the public simplify_predicates, as SimplifyExpressions folds these first.
feat: compare the groups with each other. Bounds leaving no value, and an equality that contradicts another predicate, reduce the conjunction to false; an equality drops what it subsumes; != joins the analysis and is dropped once a bound excludes its value.
test: unit tests and sqllogictest cases.

Reducing to false is valid here because a Filter keeps a row only when its predicate is true, making NULL and false interchangeable.

What is the testing strategy for this PR?

Unit tests in simplify_predicates.rs cover each comparison operator against an equality, every combination of strict and inclusive bounds, != dropped and kept, and comparisons against NULL. simplify_predicates.slt checks the plans and drops two # TODO markers this PR implements.

Are there any user-facing changes?

Filters no row can satisfy no longer scan their input. Result sets and public APIs are unchanged.

…ication `simplify_predicates` grouped every `column <op> literal` comparison by column, including ones whose literal is NULL, and then reduced each group with `ScalarValue::try_cmp`. That comparison follows sort order, where NULL is an ordinary value below every other one, rather than SQL three-valued logic. A predicate such as `a > NULL` was therefore treated as a real but weaker lower bound and dropped as redundant: a > NULL AND a > 5 => a > 5 `a > NULL` never evaluates to true, so the conjunction matches no row while the simplified `a > 5` does. Skip comparisons against a NULL literal when grouping so they are carried through untouched. Queries do not reach this today because `SimplifyExpressions` folds comparisons with NULL literals before `PushDownFilter` runs, but `simplify_predicates` is public and callers can hit it directly.

`simplify_predicates` reduced the `>`/`>=` and `<`/`<=` comparisons on a column to their most restrictive bound, but never compared the two groups with each other, and only looked for contradictions between equalities. Conjunctions that no row can satisfy were therefore left in the plan, and comparisons already implied by an equality were still evaluated per row. Reason across the groups instead, taking the same approach DuckDB's `FilterCombiner::AddFilter` does: - Contradicting bounds reduce the conjunction to `false`, so that `EliminateFilter` and `PropagateEmptyRelation` can prune the plan they filter. `x > 6 AND x < 5` is unsatisfiable, and so is `x > 1 AND x < 1` because a strict comparison excludes the value the bounds share. Note that DuckDB stops short of the latter. - An equality pins the column to a single value, so it subsumes every other predicate that value satisfies, and contradicts the rest: `x = 5 AND x > 3` becomes `x = 5`, while `x = 5 AND x > 5` becomes `false`. - `!=` predicates now take part in the analysis. One is dropped once a bound already excludes its value, as in `x > 10 AND x != 5`, and one that contradicts an equality reduces the conjunction to `false`. A column whose predicates contradict each other now short circuits the whole list, since predicates on other columns cannot make the conjunction true again. `false` stands for a conjunction that never evaluates to true, which under three-valued logic includes evaluating to NULL. That is only equivalent for the predicates of a `Filter`, which keeps a row solely when they evaluate to true, and is where this runs.

Add unit tests for the reasoning that spans the comparison groups of a column, and sqllogictest cases that check the plans it produces: - an equality subsuming the predicates its value satisfies, and being rejected by each of the six comparison operators; - bounds that leave no value, for every combination of strict and inclusive comparisons, next to the inclusive pair that admits one; - one column's contradiction discarding the predicates on other columns; - `!=` being dropped once a bound excludes its value, and kept otherwise; - comparisons against a NULL literal staying untouched, which only a unit test can reach since `SimplifyExpressions` folds them beforehand.

wudidapaopao · 2026-09-11T20:46:14Z

Hi! This is my first contribution, so the CI workflows need a committer's approval to run. Could someone help trigger them?
cc @alamb @adriangb

codecov-commenter · 2026-09-11T21:40:15Z

Codecov Report

❌ Patch coverage is 91.02564% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.95%. Comparing base (6079a1e) to head (4fbd6a6).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...er/src/simplify_expressions/simplify_predicates.rs	91.02%	3 Missing and 11 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #25207    +/-   ##
========================================
  Coverage   81.95%   81.95%            
========================================
  Files        1133     1133            
  Lines      423799   423924   +125     
  Branches   423799   423924   +125     
========================================
+ Hits       347307   347424   +117     
  Misses      55899    55899            
- Partials    20593    20601     +8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Grouping `!=` predicates by column moves them behind the ones that stay ungrouped, so `p_brand != 'Brand#45'` now follows `p_size IN (...)` in the conjunction. The predicates themselves are unchanged.

wudidapaopao · 2026-09-12T08:53:28Z

Updated the TPC-H q16 plan. The filter changed from

p_brand != 'Brand#45' AND p_size IN (...) AND p_type NOT LIKE 'MEDIUM POLISHED%'

to

p_size IN (...) AND p_brand != 'Brand#45' AND p_type NOT LIKE 'MEDIUM POLISHED%'

simplify_predicates now groups NotEq by column, and grouped predicates are emitted after the ungrouped ones, so p_brand != 'Brand#45' no longer comes first. The predicates themselves are unchanged.

wudidapaopao · 2026-09-12T17:41:45Z

Merged main to pick up #25216, which fixes the cargo test datafusion-cli failure. The failure was unrelated to this PR.

jayzhan211 · 2026-09-13T00:34:53Z

@wudidapaopao Hi, thanks for your contribution. Although this PR may solve the simple predicate you mentioned but I hope we could come out a more general framework that works for complex predicate as well. Would you like to work on it?

jayzhan211

Nested case is not covered

Wrong results for struct and list literals

satisfies_all / is_empty_range drop or contradict predicates based on ScalarValue::try_cmp. For nested types that ordering doesn't match what BinaryExpr does at runtime (compare_op_for_nested uses make_comparator with SortOptions::default(), so NULLs sort first):

partial_cmp_struct skips NULL fields, so {a: NULL, b: 1} compares Equal to {a: 2, b: 1}.
partial_cmp_list puts NULL elements above non-NULL values; runtime puts them below.

Both reproduce on this branch:

CREATE TABLE l AS SELECT make_array(arrow_cast(NULL,'Int64')) AS c;
SELECT * FROM l WHERE c = make_array(arrow_cast(NULL,'Int64'))
                  AND c < make_array(arrow_cast(1,'Int64'));
-- 0 rows / EmptyExec, but both conjuncts evaluate to true for the row

CREATE TABLE s2 AS SELECT named_struct('a', arrow_cast(2,'Int32'), 'b', 1) AS c;
SELECT * FROM s2 WHERE c = named_struct('a', arrow_cast(NULL,'Int32'), 'b', 1)
                   AND c = named_struct('a', arrow_cast(2,'Int32'), 'b', 1);
-- returns {a: 2, b: 1}; the first conjunct is false for that row

The struct case is a regression: = vs = used to be checked with structural == and correctly became false.

I'd only group predicates whose literal type's ordering matches the kernels, and treat an uncomparable pair as "leave alone" rather than failing the query:

                 ) && !is_null(&left)
-                    && !is_null(&right) =>
+                    && !is_null(&right)
+                    && right
+                        .as_literal()
+                        .or_else(|| left.as_literal())
+                        .is_some_and(|v| !v.data_type().is_nested()) =>

 let mut result = other_predicates;
 for (_, preds) in column_predicates {
-    let simplified = simplify_column_predicates(preds)?;
+    // Literals of one column that can't be ordered against each other carry
+    // no information we can use, so keep them as written
+    let simplified = match simplify_column_predicates(preds.clone()) {
+        Ok(simplified) => simplified,
+        Err(_) => preds,
+    };

Please also add the two queries above to simplify_predicates.slt so this stays covered.

wudidapaopao added 2 commits September 12, 2026 04:21

github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Sep 11, 2026

wudidapaopao force-pushed the simplify-predicates-contradictions branch from e4043ff to 9fb1c19 Compare September 11, 2026 20:31

test(optimizer): update TPC-H q16 plan for the new predicate order

5cb8243

Grouping `!=` predicates by column moves them behind the ones that stay ungrouped, so `p_brand != 'Brand#45'` now follows `p_size IN (...)` in the conjunction. The predicates themselves are unchanged.

Merge branch 'main' into simplify-predicates-contradictions

4fbd6a6

jayzhan211 reviewed Sep 13, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: simplify contradicting and redundant predicates on a column - #25207

feat: simplify contradicting and redundant predicates on a column#25207
wudidapaopao wants to merge 5 commits into
apache:mainfrom
wudidapaopao:simplify-predicates-contradictions

wudidapaopao commented Sep 11, 2026 •

edited

Loading

Uh oh!

wudidapaopao commented Sep 11, 2026

Uh oh!

codecov-commenter commented Sep 11, 2026 •

edited

Loading

Uh oh!

wudidapaopao commented Sep 12, 2026 •

edited

Loading

Uh oh!

wudidapaopao commented Sep 12, 2026

Uh oh!

jayzhan211 commented Sep 13, 2026

Uh oh!

jayzhan211 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wudidapaopao commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

wudidapaopao commented Sep 11, 2026

Uh oh!

codecov-commenter commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wudidapaopao commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wudidapaopao commented Sep 12, 2026

Uh oh!

jayzhan211 commented Sep 13, 2026

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wudidapaopao commented Sep 11, 2026 •

edited

Loading

codecov-commenter commented Sep 11, 2026 •

edited

Loading

wudidapaopao commented Sep 12, 2026 •

edited

Loading